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Abstract. [Purpose] To examine agreement rates between faculty members and clinical supervisors as OSCE 
examiners. [Subjects] The study subjects were involved physical and occupational therapists working in clinical 
environments for 1 to 5 years after graduating from training schools as OSCE examinees, and a physical or oc- 
cupational therapy faculty member and a clinical supervisor as examiners. Another clinical supervisor acted as a 
simulated patient. [Methods] The agreement rate between the examiners for each OSCE item was calculated based 
on Cohen's kappa coefficient to confirm inter-rater reliability. [Results] The agreement rates for the behavioral as- 
pects of the items were higher in the second than in the first examination. Similar increases were also observed in 
the agreement rates for the technical aspects until the initiation of each activity; however, the rates decreased during 
the middle to terminal stages of continuous movements. [Conclusion] The results may reflect the recent implementa- 
tion of measures for the integration of therapist education in training schools and clinical training facilities. 
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INTRODUCTION 

In Japan's super-aging society, social demands for reha- 
bilitation are increasing. In line with this, the numbers of 
psychical therapist (PT) and occupational therapist (OT) 
training schools have rapidly increased to 248 and 182, re- 
spectively, and the total numbers of certified PTs and OTs 
were 110,000 and 65,000, respectively, as of 2013. The 
number of therapists is particularly excessive in the field 
of physical therapy, as more than 10,000 novices are certi- 
fied on an annual basis. This has resulted in a high propor- 
tion, approximately 25%, of novice PTs and OTs with less 
than 3 years of clinical experience. The rapid increase in 
the number of novice therapists has consequently reduced 
overall years of experience of therapists in clinical envi- 
ronments, suggesting a decrease in the quality of clinical 
services. Under these circumstances, it may be necessary 
to expand the range of therapists' activities. Also, in order 
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to improve the conditions of their employment, which have 
become unfavorable over the past years, it may be crucial 
to enhance novice therapists' abilities, while continuously 
training experienced therapists and nurturing specialists. 

Diverse approaches to post-graduate training have al- 
ready been implemented in clinical environments; however, 
such approaches have not been unified, and appropriate 
methods for assessing novice therapists' clinical skills, as 
well as educational methodologies, have not yet been fully 
examined. Considering this situation, the unification of 
post-graduate and clinical education methods, based on the 
results of objective assessment of clinical skills in the early 
stages, may be vital for novice therapists' development. 

At our university, physical and occupational thera- 
pies are regarded as a domain of therapeutics, or a clini- 
cal science, and specialized education for students, in 
other words, "education to nurture specialists with clinical 
skills", has been provided, focusing on clinical demands, 
since our department was founded in 2004. In short, our 
education focuses on clinical skills. However, in the current 
therapist education system, students' clinical skills are as- 
sessed by supervisors of clinical training facilities, rather 
than the faculty members of training schools. In therapist 
education to nurture "clinical professionals", this is a very 
important issue. Our faculty members are also engaged in 



1148 J. Phys. Ther. Sci. Vol. 26, No. 8, 2014 



clinical treatment, while continuously discussing appropri- 
ate methods for training students with therapists working 
in clinical environments. At our university, the Objective 
Structured Clinical Examination (OSCE) system has been 
adopted, with a view to specifying standards for clinical 
education in training schools and standardizing methods of 
assessment by clinical supervisors''. The OSCE is a method 
of clinical skill assessment, which was proposed by Hard- 
en^' in 1975, and it has been reported to be appropriate for 
assessing learning achievement levels in the psychomotor 
and emotional domains, which are difficult to evaluate with 
written examinations. 

Aiming to establish an OSCE-based education system, 
we have conducted the OSCE before clinical training since 
2005. The examination has also been conducted after clini- 
cal training since 2008, when the OSCE items were adopted 
as those for practice (so that students can systematically 
learn about each item, and assimilate the learning with due 
consideration for the behavior and skills needed in clinical 
environments as a preparatory step for clinical training). 

We reported an association between OSCE scores, aca- 
demic achievement levels, and clinical training outcomes, 
as well as inter-rater agreement rates (0.767 to 0.935) be- 
tween faculty members in previous studies^^^l The present 
study focused on the necessity of confirming consistency 
between OSCE scores, representing the results of assess- 
ment by faculty members and clinical training outcomes as- 
sessed by clinical supervisors, in order to adopt the OSCE 
in post-graduate education. The objectivity of assessment 
refers to the consistency of results between different raters, 
and, therefore, it may be important to enhance the objectiv- 
ity of OSCE-based assessment in terms of accountability 
of the assessment results. Accordingly, this study examined 
agreement rates between faculty members and supervisors 
of clinical training facilities as OSCE examiners. 

SUBJECTS AND METHODS 

The study subjects were PTs and OTs working in clinical 
environments for 1 to 5 years after graduating from train- 
ing schools as OSCE examinees, and a physical or occupa- 
tional therapy faculty member and a clinical supervisor as 
raters. Another clinical supervisor acted as a simulated pa- 
tient. Before examination, the examiners (the faculty mem- 
ber and clinical supervisor) and simulated patient agreed 
upon the latter's condition, in addition to the points of as- 
sessment. 

The OSCE consisted of items to assess skills to assist 
and guide patients performing the following level 3 activi- 
ties: elevating the upper body and sitting on the edge of the 
bed; standing up; wheelchair-to-bed transfer; dressing; toi- 
leting; and walking. It was conducted before (examination 
1) and after (examination 2) learning sessions consisting of 
60-minute lectures regarding each item, practice, and prac- 
tical guidance in clinical environment, focusing on the ap- 
plicability of the OSCE items in clinical environments. 

The OSCE was conducted in 3 rooms (stations), in each 
of which a task was presented, and the examinees made the 
rounds of these stations to perform all tasks, following in- 



structions. At each station, 2 examiners (a physical or occu- 
pational therapy faculty member and a clinical supervisor) 
and 1 simulated patient (another clinical supervisor) were 
present. The time allowed for performance of each task was 
5 minutes, and, immediately after task completion, 2 min- 
utes of feedback was provided. Subsequently, the examiners 
and simulated patient discussed and assessed the examin- 
ees' performance. The tasks were not previously shown to 
the examinees. 

For statistical analysis. Predictive Analytics Software 
(PASW) Statistics 18.0 was used. To confirm inter-rater 
reliability, the agreement rate for each item was calculat- 
ed based on Cohen's kappa coefficient. After calculation, 
means were adopted as data regarding the behavioral and 
technical items. 

RESULTS 

The agreement rates between the examiners for the be- 
havioral aspects of transfer, dressing, standing up, and toi- 
leting were high in the examination. In contrast, those for 
elevating the upper body and sitting on the edge of the bed 
and walking were low. In examination 2, the agreement rate 
was 0.7 or higher for all items. For the technical aspect of 
transfer, the agreement rate was the highest, 0.5, in exami- 
nation 1, followed by toileting, standing up, dressing, and 
walking, with rates of 0.3 to 0.4. The agreement rate was 
the lowest for elevating the upper body and sitting on the 
edge of the bed, 0.25. In examination 2, the agreement rates 
were generally higher than those in examination 1; 0.4 to 
0.6 for transfer, standing up, toileting, and walking, and 0.3 
for elevating the upper body and sitting on the edge of the 
bed and dressing (Table 1). 

The agreement rate was significantly low for the techni- 
cal aspects of the following sub-items: elevating the upper 
body and sitting on the edge of the bed, being able to moti- 
vate the patient to rise and enhance his/her activeness, be- 
ing able to confirm sufficient space for the lateral position, 
being able to appropriately guide the lower body toward the 
lateral position and being able to give instructions at appro- 
priate times throughout the activity and dressing, being able 
to take appropriate safety measures, being able to confirm 
appropriate procedures before placing the patient's arms 
in a sleeve, being able to provide appropriate guidance for 
placing the patient's paralyzed arm in a sleeves, being able 
to provide appropriate guidance for placing the patient's 
non-paralyzed arm in a sleeve (while ensuring a stable sit- 
ting position), being able to provide appropriate guidance/ 
assistance for the elbow and shoulder when placing the par- 
alyzed arm in a sleeve, and prepare for pulling an upper gar- 
ment over the head and being able to correct the shoulder 
line and bottom edge of the garment on the paralyzed side. 
Furthermore, the agreement rates for the technical aspects 
remained high until the initiation of each activity, and, de- 
creased during the middle to terminal stages of continuous 
movements (Table 2). 
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Table 1. Intra-class corrclalion coefficients For the facully members and cHnical superv isors 



Sitting up 
on the bed 


Standing up 


Transfer: wheel- 
chair-to-bed transfer 


Toileting 
activity 


Dressing 
activity 


Walking 


Preparatory items 












1st 0.534 


1.000 


0.860 


0.910 


0.738 


0.399 


2nd 1.000 


1.000 


0.911 


0.867 


1.000 


0.745 


Technical items 












1st 0.245 


0.429 


0.520 


0.430 


0.380 


0.341 


2nd 0.294 


0.566 


0.630 


0.487 


0.286 


0.469 



Table 2. Agreement rales between the examiners in the OSCE lc\cl 3 (examinations 1 and 2) 



Sitting up on the bed 


1st 


2nd 


Preparatory items 






• Appropriately groomed 


1.000 


1.000 


• Being able to appropriately greet 


0.446 


1.000 


• Being able to provide appropriate explanations before assessing movements to elevate the upper body and sit on 
the edge of the bed 


-0.149 


1.000 


• Being able to appropriately confirm the patient's consent 


1.000 


1.000 


• Being able to appropriately remove the bed rail by the patient's leg 


0.463 


1.000 


• Being able to remove and keep blankets and clothes on the bed in order 


0.383 


1.000 


Technical items 






• Being able to motivate the patient to rise and enhance his/her activeness 


0.267 


0.007 


• Being able to confirm sufficient space for the lateral position 


0.285 


-0.048 


• Being able to roughly determine the patient's remaining functions 


0.474 


0.565 


• Being able to provide appropriate explanations regarding procedures from elevating the upper body to sitting on 

the edge of the bed 


O.HJU 




• Being able to appropriately guide the head upward 


0.092 


0.364 


• Being able to appropriately guide the upper body toward the lateral position 


0.022 


0.718 


• Being able to appropriately guide the lower body toward the lateral position 


0.110 


0.182 


• Being able to give instructions at appropriate times throughout the activity 


0.178 


-0.004 


• Being able to stand in an appropriate position for guiding the patient 


0.162 


0.371 


• Being able to take preventive measures against falls, while maintaining an appropriate sitting position 


0.414 


0.353 


Transfer: wheelchair-to-bed transfer 


1st 


2nd 


Preparatory items 






• Appropriately groomed 


1.000 


0.645 


• Being able to appropriately greet 


1.000 


1.000 


• Being able to provide appropriate explanations before assessing transfer 


1.000 


1.000 


• Being able to appropriately confirm the patient's consent 


0.450 


1.000 


Technical items 






• Being able to appropriately place the wheelchair by the bed 


0.130 


0.645 


• Being able to adjust the height of the bed to ensure safety 


0.743 


0.564 


• Being able to advise or assist the patient to raise the foot plates 


0.083 


1.000 


• Being able to advise the patient to participate in the activity as much as possible 


0.593 


0.805 


• Being able to provide guidance for moving the buttocks forward and sitting on the anterior part of the wheelchair 


0.394 


0.273 


seat 


• Being able to place the non-paralyzed foot slightly ahead of the paralyzed one 


0.031 


0.434 


• Being able to support the patient's knee on the paralyzed side by maintaining it between the distal parts of both 
thighs 


0.586 


0.732 


• Being able to give advice for bending the trunk forward, while placing the hands by the waist or on the knees 


0.450 


0.583 


• Being able to support the patient's chest and abdomen with the forearms before standing up 


0.700 


0.538 


• Being able to give instructions to stand up together at an appropriate time 


0.911 


0.563 


• Being able to guide/assist the patient to turn his/her buttocks to the bed, and slowly sit on it 


0.901 


0.596 


• Being able to correct the posture when sitting on the edge of the bed, and make it stable 


0.705 


0.829 
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Dressing activity 



1st 2nd 



1.000 1.000 

1.000 1.000 

1.000 1.000 

-0.048 1.000 



Preparatory items 

• Appropriately groomed 

• Being able to appropriately greet 

• Being able to provide appropriate explanations before assessing dressing 

• Being able to appropriately confirm the patient's consent 
Technical items 

• Being able to take appropriate safety measures 

• Being able to confirm the maintenance of a stable sitting position 

• Being able to confirm the reach movements of the non-paralyzed upper body 

• Being able to confirm appropriate procedures before placing the arms in the sleeves 

• Being able to provide appropriate guidance for maintaining the paralyzed arm straight by the trunk, using the 
weight of the upper body 

• Being able to provide appropriate guidance for placing the paralyzed arm in the sleeves 

• Being able to provide appropriate guidance for placing the non-paralyzed arm in the sleeves (while ensuring a 
stable sitting position) 

• Being able to provide appropriate guidance/assistance for the elbow and shoulder when placing the paralyzed 
arm in the sleeves, and prepare for pulling an upper garment over the head 

• Being able to provide appropriate guidance/assistance for drawing the garment over the head (while maintaining 
the pelvis in a neutral position) 

• Being able to correct the shoulder line and bottom edge of the garment on the paralyzed side 
Standing up 



0.419 
0.574 
0.384 
0.532 

0.160 

0.220 

0.211 



-0.253 
0.554 
1.000 

-0.065 

0.680 
0.000 
-0.078 



0.450 -0.182 



0.516 0.974 



0.333 
1st 



0.227 
2nd 



1.000 1.000 

1.000 1.000 

1.000 1.000 

1.000 1.000 



Preparatory items 

• Appropriately groomed 

• Being able to appropriately greet 

• Being able to provide appropriate explanations before assessing movements to stand up 

• Being able to appropriately confirm the patient's consent 
Technical items 

• Being able to confirm the appropriate height of an adjustable bed 

• Being able to confirm the position of the patient's buttocks, and move them forward with guidance/assistance 

• Being able to move the patient's legs at an appropriate angle, confirm the appropriate width between his/her feet 
and heels' condition on the floor, and correct them with guidance/assistance 

• Being able to correct the posture, alignment, and center of balance with guidance/assistance 

• Being able to confirm and give advice for anterior pelvic tilt and participation as a preparatory step for leaving 
the bed 

• Being able to support the patient's pelvis with the hands and his/her trunk with the chest, while standing with the 
knees bent when leaving from the bed 

• Being able to move the patient's knees forward using the legs when leaving from the bed 

• Being able to bend the patient's head and trunk forward, and shift the center of balance at a safe speed with guid- 
ance/assistance 

• Being able to support the patient's knees, and ensure appropriate hip, knee, and trunk extension with guidance/ 
assistance after leaving from the bed 

• Being able to correct the alignment with guidance/assistance to maintain a stable standing position 



0.471 


1.000 


0.093 


1.000 


0.455 


0.241 


0.130 


0.669 


0.538 


0.739 



0.193 


0, 


,364 


0.492 


0. 


,323 


0.773 


0, 


,353 


0.637 


0, 


,336 


0.505 


0, 


,630 



Toileting activity 
Preparatory items 

• Appropriately groomed 

• Being able to appropriately greet 

• Being able to provide the patients with appropriate explanations before toilet training 

• Being able to appropriately confirm the patient's consent 

Technical items 

• Being able to contirm appropriate environments for the activity 

• Being able to provide appropriate explanations regarding procedures 

• Being able to provide guidance for appropriately standing up from the bed 

• Being able to provide guidance for a change of direction and stable standing position 



1st 



2nd 



0.645 1.000 

1.000 1.000 

1.000 1.000 

1.000 0.463 

0.621 0.776 

0.669 0.000 

0.405 0.441 

0.114 0.389 
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Table 2. Conlinuc 



• Being able to confirm the appropriate posture when puUing down clothes 


U.Ul J 


U.4/0 


• Being able to give advice for smoothly and appropriately handling clothes 








0.385 


0.719 


• JJCIIlg dUlC lU JJlUVlUC gUlUdilL'C lUl IIIC lIldlUlCIldllL'C Ui dll dJjpHJJJI IdlC OlLllUg jJUOlLIUII 


0.281 


0.448 


tt RpiTiD" anlp tn oivp anvipp for JitiTirnnriJitplv plpaninp trip ami^/D'PTiitjil'; 


0.321 


0.506 


M RpintT nnlp tr\ twfW7Mtf^ (TiiiHcinpp tt^i" nnr^irtnT'iQfpl^/ ctnTiriintT iiri TiT^m frip tnilpt cpnt 
• JJCIIl^ dUIC lU JJiUVlUC ^UlUdli^C IXJL djJjJiUpi idLCiy aLdilUIIl^ U]J llUlil LliC LUilCl oCdl 


0.551 


0.296 


• DCIIlg dUIC lU L'UlIIIlill IIIC djJJJlUpi Idle pUblUlC WIICIl JJUlIiIIg U]J CIULUCS 


0.486 


0.700 


• Bemg able to give advice for smoothly and appropriately handling clothes when pulling up them 


n Qi 9 


n '^01 

v.jyL 


• JjCIII^ dUIC LU piUVlUC ^UlUdliCC iUl a ClldligC UI UilCL-LIUll ailU DiLLllIg Ull LIIC CUgC Ui lliC UCU 






VV diJtlllg 


let 




I^lCpalalUiy ILClIio 






Annrnnnntplv (rrnnmpri 


0.645 


1.000 


Rpino" ahlp tn annrnnriJitplv orppt 


-0.048 


1.000 


• ociiig duic lu piuviuc dppiupiidLC cApidiidiiuiio uciuic pci iUl Illlllg gd^l dlldiyalo 


0 nno 




# Rpino nblp tn amiT'n'nfifltplv caniiTm tTiP nntiptit's; pnnQPtit 

~ ILl^ ClULs^ VVJ CLIJIJI \JVJL IcHCl y Vv^XlXll. 1X1 Lll^ L/ClI'ld-i-l' 0 wL'llOd-i-l' 


1.000 


1.000 


Technical items 






• Being able to take safety measures 


o./yy 


A 1 on 


• Bemg able to ensure appropriate walking environments 


-0.038 


0.463 


• Being able to confirm appropriate clothes, shoes, and aids for walking 


0.284 


0.349 


• Being able to advise the patient to participate in the activity as much as possible 


-0.048 


0.369 


• Being able to confirm the ability to move in a standing position 


0.000 


0.542 


• Being able to appropriately instruct to start walking 


0.753 


0.780 


• Being able to appropriately provide guidance/assistance during the swing phase on the paralyzed side 


0.741 


0.441 


• Being able to appropriately provide guidance/assistance during the initial contact to loading-response phases on 

the paralyzed side 


0.593 


0.545 


• Being able to appropriately provide guidance/assistance during the mid-stance phase on the paralyzed side 


0.094 


0.450 


• Being able to appropriately provide guidance/assistance during the terminal stance phase on the paralyzed side 


0.000 


0.567 


• Being able to confirm appropriate walking rhythms 


0.577 


0.535 


• Being able to appropriately guide the patient after walking 


0.831 


0.411 



DISCUSSION 

In recent years, the OSCE has been used as an education- 
al approach to the objective assessment of clinical skills, 
mainly in the field of medicine^"^'. Compared to conven- 
tional written examinations, the OSCE enables examiners 
to assess clinical skills in the psychomotor, emotional, and 
cognitive domains, and clarify points to improve^'. Saitoh et 
al. reported that the adoption of the OSCE in medical educa- 
tion has been effective at training medical students through 
the development of necessary basic skills in both techni- 
cal and behavioral aspects, and it also enables educators to 
guide students in the appropriate integration of knowledge, 
skills, and behavior''. In fact, the OSCE adopted for clini- 
cal education in training schools has provided a certain ef- 
fect^"^'; however, education methods in training schools 
and clinical training facilities have not yet been unified. 

Considering this situation, the present study examined 
agreement between faculty members and clinical super- 
visors of OSCE-based assessment results, with a view to 
adopting the OSCE system in post-graduate education. 

In examination 1, the agreement rates between the exam- 
iners for transfer, dressing, standing up, and toileting were 



high in the behavioral aspect, while those for elevating the 
upper body and sitting on the edge of the bed and walking 
were low. In contrast, in examination 2, a high agreement 
rate of 0.7 or more was obtained for all items, suggesting 
improvement. In the technical aspects, the agreement rates 
were generally lower in both examinations 1 and 2 than 
those of behavioral aspects. The rate was particularly low, 
approximately 0.3, for elevating the upper body and sitting 
on the edge of the bed and dressing. Comparison of exami- 
nations 1 and 2 revealed the agreement rates were generally 
higher in the latter than in the former. 

The agreement rates for the behavioral aspects of elevat- 
ing the upper body and sitting on the edge of the bed, trans- 
fer, dressing, and standing up increased in examination 
2. Nakahara et al. reported that, in the field of rehabilita- 
tion, including physical and occupational therapies, clini- 
cal supervisors frequently point out students' problems in 
the emotional domain, such as insufficient sociability and 
inappropriate basic behavior'"'. The high agreement rates 
between the faculty member and clinical supervisor in the 
present study suggest that our OSCE-based assessment sys- 
tem may have a sufficient inter-rater reliability. On the other 
hand, the agreement rate for walking was lower than those 



1152 J. Phys. Ther. Sci. Vol. 26, No. 8, 2014 



for other items. This may be explained by a tendency for 
this activity to be regarded as PTs' specialty, which may 
have influenced the results of this study as both PT and 
OT were the subjects. This supposition is supported by the 
markedly low agreement rate of the sub-item, being able 
to provide appropriate explanations before performing gait 
analysis, which may directly represent a specialty differ- 
ence between the examiners. 

For the technical aspects, the agreement rates were gen- 
erally lower than those of the behavioral aspects. Markedly 
low rates of 0.2 to 0.3 were observed for elevating the up- 
per body and sitting on the edge of the bed, and dressing. 
Particularly, for the sub-items of the former (being able to 
motivate the patient to rise and enhance his/her activeness, 
being able to confirm sufficient space for the lateral posi- 
tion and being able to give instructions at appropriate times 
throughout the activity) and the latter (being able to take 
appropriate safety measures and being able to confirm ap- 
propriate procedures before placing the arms in the sleeves), 
which represent the technical and partly behavioral aspects 
of preparation and communication skills, agreement was 
rarely obtained. Conversation rhythms, as well as the tone 
and volume of voice, are important factors for communi- 
cation skills; therefore, the agreement rates for items rep- 
resentative of their technical aspects may have been influ- 
enced by examiners' subjectivity more markedly than those 
of other items related to assistance and guidance. Similarly, 
the low agreement rate for elevating the upper body and sit- 
ting on the edge of the bed, an activity requiring continuous 
movements, may be explained by differences in from the 
method taught to perform this activity, taught in schools, in 
which the movements are separately analyzed. In short, in 
the OSCE, it may be necessary to consider the influence of 
examiners' sense of values and the consequent subjectivity 
of their assessment. To address this issue, paradigm-based 
training for examiners and discussions regarding possible 
problems may be needed. To improve the inter-rater agree- 
ments, items that involve a mix of contents should be sub- 
divided. In addition, in low agreement rate items, clear cri- 
teria for each rating should be decided by discussion among 
the rates. 

The results of this study, may reflect the recent imple- 



mentation of measures for the integration of therapist edu- 
cation in training schools and clinical training facilities. 
Rehabilitation medicine is a practical system, in which 
medical services are provided based on learning achieve- 
ments. Therefore, in therapist education aiming to nurture 
learning specialists, it may be important to establish OSCE- 
based assessment systems, while reviewing the significance 
of OSCE -based education from the viewpoint of rehabilita- 
tion medicine, to devise new approaches, rather than simply 
adopting medical education systems. Further studies may 
be necessary to develop systems for the adoption of the 
OSCE in the training of novice therapists working in clini- 
cal environments and to provide lifelong education. 
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